Import Metadata

This topic is related to the following section - Adding & Editing Datasources.

About Import Metadata

The Syniti Data Catalog is the system of record for all datasources stored as system datasources within Syniti Migrate. The import of metadata from the legacy systems to the Data Catalog will pull in all tables and rows of all databases within that relevant Datasource. Should a value change or a new field be added to the table within a legacy system, the Data Catalog is refreshed to update and then this data will be available to import to Syniti Migrate through the Import Metadata process.

Within Datasources, while viewing Datasource Table Details, click Options  icon and click Import Metadata to refresh the metadata from SKP Data Catalog for changes to tables, fields, or columns, and for adding new metadata.

This button allows users to refresh the active tables and columns and show the most current metadata stored within SKP Data Cataog.

Should the datasource have a reasonable list of tables to pull, then the user may toggle Import All as active which pulls every table in the SKP Data Catalog list. In the case of systems with large numbers of tables, the user should build out a list of valid, active tables within Administer > Advanced > Applications. This subset of tables are pulled into the import metadata job queue when Import Metadata is selected at the Datasource level. 

Note

If the data imports from an SAP system, the Import All toggle is hidden. Setup Applications for SAP with all needed Active tables, and then run Import Metadata from the Datasource. Refer to section Administer > Advanced > Applications for more details of this process.

Note

During the initial Import of Metadata for an Application, only the active tables are added to the import.Metadata for all tables (including System tables) imports in all subsequent runs.

Datasources: Details: Import Metadata

The Import All toggle in the Metadata section of details overrides the filters setup in Administer > Advanced > Applications to import all tables and fields.

Note

Should the Import All toggle be inactive and there are no tables assigned to the Datasource within Applications, a message shows that no metadata is imported for this situation.

To schedule a repeated refresh, run the refresh and then setup as a Scheduled Job within Job Queues.Refer to section Monitor > Job Queues for details of this process.

The user selects Import Metadata to initiate the import. A warning message displays. Click the OK button within the Deploy window to continue. The message "Import metadata job has been successfully added to job queue" displays once set.

Datasource: Import Metadata: Warning message

When a user attempts to import the metadata of a datasource, the system may prompt them to confirm the action as shown below. This can happen both as part of the datasource wizard, and also when they click the "Import Metadata" option on the datasource form's ellipsis menu.

Datasource: Import Metadata: Warning message

Import Metadata Job Queue

If the Datasource has a valid connection setup, the system executes an update to save a new Job for the import of this Datasource based upon the options selected within the metadata section as shown below.To view details of the job, use pathway Monitor > Job Queues.

The details of the Import datasource job and Status should be available to view within the Job Queue page. If there are errors, fix the connection and try again.

Administer: Job Queue: Import Datasource metadata

Once the Job is In Process, the tables as they are building are added to the details for Table.Click Debug icon within panel Steps to view the table details as shown below.

Administer: Debug Log: Import Table Details

Job Queue details for Import Metadata is a vertical split page with two panels of data - Tables on the left side and Columns or Constraints tabs on the right side. The Tables panel provides the list of all tables that were included in the Metadata import effort. If the Refresh icon is active, then the Datasource may be imported again using this button.

Refer to section Monitor > Job Queues for section titled Importing Table Data to a Datasource for more details of this process.

If the Datasource contains an Invalid Connection, then the process fails immediately with a message as shown below:

Datasource: Import Metadata: Error for Invalid Connection

Import Metadata at the Table level

The user may also refresh one table at a time by clicking the Edit icon to view the Table details as shown below. Click the Options  icon and select Import Metadata to initiate the job queue to refresh the table and columns for the selected table.

Note

Should the selection for Import Metadata be grayed out, check the details of this table. Most likely the Schema* field is blank as it is not necessary for Import all, but for individual table import, it needs to have a value.

Datasources: Edit Table: Import metadata

The user clicks Import metadata button to initiate the import. A warning message displays. Click the OK button within the Deploy window to continue. The message "Import metadata job has been successfully added to job queue" displays once set.

Datasource: Import metadata: Warning message

The system execute an update to save a new Job for the import of this Table. The details of the Import table job and status should be available to view within the Job Queue page. If there are errors, fix the connection and try again.

Administer: Job Queue: Import Table metadata

Once In Process, the table data stores within the panel for Steps. Click Debug icon to view the table details.

Administer: Debug Log: Table Details

Import Metadata Response to Added Columns in a Table

Should a column be added manually to a System Datasource table for use in Mapping (Refer to Catalog > Datasources > Adding & Editing Datasources > Adding & Editing Tables or Columns for details of adding new columns to existing tables in a System Datasource) the Import of Metadata may require updates made to the added field. 

In the process of importing the metadata for a table, Syniti Migrate checks that the fields of that table match exactly - same count and same names. Should there be an added field in the target table for System Datasource that does not exist within the Target system, Syniti Migrate updates to mark that field as Deleted and a comment is added to the Comments field as well.

In the example below, the field zADDMANUAL exists in the target table with the Syniti Migrate Database, but not within the actual system datasource table. To prevent the process from failing, this field is set as Do Not Import and the system ignores the field in the XML Build for the Dataflow. The tables show the field, but there is no import of data for that field. 

Systems Datasource Table: Added Column Marked as Do Not Import